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Abstract — The excellent performance of convolutional low- 
density parity-check codes is the result of the spatial coupling of 
individual underlying codes across a window of growing size, but 
much smaller than the length of the individual codes. Remark- 
ably, the belief-propagation threshold of the coupled ensemble 
is boosted to the maximum-a-posteriori one of the individual 
system. We investigate the generality of this phenomenon beyond 
coding theory: we couple general graphical models into a one- 
dimensional chain of large individual systems. For the later we 
take the Curie- Weiss, random field Curie-Weiss, .ff-satisnability, 
and Q-coloring models. We always find, based on analytical 
as well as numerical calculations, that the message passing 
thresholds of the coupled systems come very close to the static 
ones of the individual models. The remarkable properties of 
convolutional low-density parity-check codes are a manifestation 
of this very general phenomenon. 

I. Introduction 

Convolutional low-density parity-check (LDPC) codes ini- 
tialy introduced by Felstrom and Zigangirov [T| have been 
recognized to have excellent performance, and have spurred a 
large body of work (see |2], Gl, |4|, [5| and references in [6]). 
A complete mathematical analysis of the mechanism which 
operates behind these constructions has been achieved recently 
|6| for the binary erasure channel (BEC). Convolutional LDPC 
ensembles are constructed by coupling together, across a 
window of finite width, copies of a standard individual LDPC 
ensemble into a one dimensional chain. In order to sucessfully 
decode, one typicaly "terminates" the chain by assuming that 
the codebits at the two ends of the chain are known to the 
decoder. 

The natural thresholds involved are £bp, £map for the belief 
propagation (BP) and maximum-a-posteriori (MAP) decoders 
of the individual LDPC ensemble and the ones of the coupled 
ensemble eg°p plod , e^p 6 ■ When the length of the chain 
tends to infinity, £map° ~~ ^ £ map from above. But the main 
reason for the success of convolutional LDPC ensembles is 
that as the width of the window increases (the size of the 
individual ensemble and the length of the chain being already 
very large) eg°p pled — > £map- In fact more is true. Figure Q] 
shows (BEC channel) how the EBP EXIT curves of chain 
ensembles of various lengths, approach the MAP EXIT curve 
of the individual ensemble [6|. In the limit, the difference 
between the two curves is indistinghuishable even for a small 
coupling width. These features have also been observed, and 
partially proved, for general channels, and provide a new way 
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Fig. 1. Taken from (6). Left: EBP EXIT curves of the ensemble 
(3,6, L), L = 1, 2, 4, 128 and nearest neighbor coupling. The 
light/dark gray areas mark the interior of the BP/MAP EXIT function 
of the underlying (3, 6)-regular ensemble. Right: circle shows a 
magnified portion of the EBP EXIT curve for the (3, 6, 32) ensemble. 
The horizontal magnification is 10 3 . 



to construct capacity achieving codes. 

In this paper we show that the mechanism that is at work is 
very natural and general. We investigate a wide variety of indi- 
vidual graphical models that we couple together to form a one 
dimensional chain with appropriate boundary conditions. The 
individual models include the Curie- Weiss (CW) and random 
field Curie-Weiss (RFCW) models of statistical mechanics, 
which are defined on the complete graph, as well as constraint 
satisfaction problems such as K — SAT (K satisfiability) 
and Q — COL (Q coloring) formulated on random sparse 
graphs. As will be argued later, the underlying graph should be 
"infinite dimensional" so that the individual system displays 
a "mean field" behaviour. We provide analytical arguments as 
well as numerical evidence that in all these cases, as the width 
of the coupling increases, the message passing threshold of the 
chain comes close - in fact converges - to the static threshold 
of the individual system. 

We first discuss the coupled CW models which can be 
solved, to a large extent, by explicit analytical calculations 
and thus provide a nice pedagogic illustration of the main 
mechanisms. We then turn to the more difficult constraint 
satisfaction problems. 

II. Coupled Models on Complete Graphs 

The CW model is perhaps the simplest mean field model. 
It consists of an Ising system of TV spins Sj = ±1 attached to 
the vertices of a complete graph. The Hamiltonian function is 




Fig. 2. Van der Waals curve for the CW model at J = 1.1. The 
static phase transition threshold given by the Maxwell construction 
is h s = 0. The local maximum and minmum at heights ihit are 
iterative thresholds. 
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Fig. 3. Van der Waals curve for the coupled CW model with L = 25, 
J = 1.1 and w — 1 (nearest neighbor coupling). In the circle the 
vertical magnification is 10 4 . The iterative threshold nearly coincides 
with h 3 = 0. 



Hn(s) — s i s j where J > and the sum carries 

over all Af( -^~ 1 ^ edges (i, j) of the graph. 

The easiest way to solve this model is to consider the 
canonical ensemble in which the free energy is <l>(m) = 
— limAT^ +oc jfhiZN^m), with the canonical partition func- 
tion Zn(iti) equal to the sum of the Gibbs weights 
exp(— H (s)) over all spin configurations with their magnetiza- 
tion fixed to mN, i.e. Yli=i s i = TnN. A standard calculation 
gives (up to an irrelevant constant) <f>(m) = — -|to 2 — H(m) 
where the first term is the internal energy of the spin configura- 
tions and the second one is the contribution from their entropy 
H(m) = -i±^ln±±2* - i^ln^. Here $(m) is not 
convex because of the non local interaction, and the thermal 
equilibrium (Helmoltz) free energy is given by the convex 
hull of <f>(m). From the thermodynamic relation between the 



magnetic field and the magnetization h = d< ^Q™^ one gets the 



so-called Van der Waals curve (see figure |2) 
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which is equivalent to the CW equation m = tanh( Jm + 
h). The isotherm (i.e. the relation between the magnetic field 
and the magnetization at thermal equilibrium) is not given 
by the full Van der Waals curve but by the Maxwell equal 
area construction which yields h s — for the phase transition 
threshold. The part of the Van der Waals curve not on the 
isotherm descibes metastable and unstable states of the system. 

The CW equation can also be obtained in the grand- 
canonical ensemble where the magnetic field h is fixed and 
the total magnetization is allowed to fluctuate. One then finds 
that the Gibbs grand-potential (or "pressure") is given by a 
variational expression min m ($(m) — hm). The minimizer 
satisfies the CW equation with m now interpreted as the 
average magnetization for a fixed magnetic field h. 

One may solve iteratively the CW equation for m. When 
h > ha > (see fig. [U one finds a unique branch of 
solutions (global minimizer of $(m) — hm). For h lt > h > 
there appear two new solutions: the leftmost solution on fig. 
|2] (local minimizer) and the middle solution on fig. [2] (local 
maximizer). Physicaly the global minimizer corresponds to 
a thermal equilibrium state, and the local minimizer (resp. 
maximizer) correspond to metastable (resp. unstable) states. 
For h < the situation is symmetrical. 

The analogy with coding concepts basically goes as follows. 
The magnetic field h is a control parameter analogous to the 
channel noise e, the Gibbs grand-potential is the input-output 



entropy, the CW equation is the analog of the density evolution 
fixed point equation, the Van der Waals curve the analog of 
the EBP GEXIT curve, the iterative threshold h lt corresponds 
to 6bp and the static threshold h s = corresponds to £map- 

A. Chain of Coupled Curie-Weiss Systems 

Consider now 2L + 1 copies of the CW system labeled by 
their "positions" z. Thus at position z we have a complete 
graph with N vertices and spins Sj Z attached to them. Each 
spin at a given position z e {— L + w,...,+L — w} is 
coupled to N — 1 spins at the same position with a coupling 
of strength an< ^ to spins located at positions 

z — w, — 1, +1, z + w with couplings of strength -j-^- 
Thus the overall coupling of one spin to the rest of the system 
is J when N — > +oo, just as in the uncoupled system. 
Close to the boundary, for z 6 {— L, — L + w — 1} (resp. 
{L — w + 1, L}) we set couplings towards the left (resp. 
right) equal to zero and set those to the right (resp. left) equal 
to ofTvf. The Hamiltonian of the model can be written as 

2wN 



H N<L (s) 
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z — — L+w k—1 ij—l 

This is supplemented by a boundary condition: for z G 
{-L, —L + w — 1} (resp. {L - w + 1, L}) the local 
magnetizations are fixed to J2i s iz — m-(K) (resp. m + (h)), 
where m±(h) are the two localy stable solutions of the CW 
equation. In particular, for \h\ > h it they are equal (since (Q} 
has a unique solution) and for h = h s = they are opposite. 

This model can be solved exactly either in the canonical 
or in the grand-canonical ensembles, although the later is 
technically more convenient. Let m(z) be the average local 
magnetization m(z) = (jjJ2iLi s iz) N . L where (~)n,l is 
the average performed with the grand-canonical Gibbs weights 
exp — (Hn^l(s) + hj^i z s iz)- O ne can derive the following 
generalization of (fTJ for z = —L + w, L — w 

" = ~ ~T ~n~~2 m y z ' ~ Jm(z) + — in — - , (2) 

4 Dz A 2 1 — m[z) 

n 2 

where is a finite difference operator, 
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i(z) = — ^~^(m(z — k) — 2m(z) + m(z + k)) 



fc=l 




Fig. 4. Top left: a kink profile for L = 25, J = 1.1, h = 0; 
bars are the numerical solution m(z) (z € {—25, +25}) and the 
continuous blue line is the analytical approximation. Top right: a 
ball rolling in the inverted potential "^(m) (vertical axis), m is the 
position of the ball (horizontal axis). The initial and final velocities 
are zero and the transition from one maximum to the other takes an 
infinite time. Bottom left: profile for L — 25, J = 1.1, h — 0.55; 
bars are the numerical solution. The continuous blue curve is an 
analytical approximation valid for z >> — L, the green segment an 
analytical approximation valid for z near — L. Bottom right: the ball 
starts from the local maximum with enough initial kinetic energy in 
order to climb on the right maximum of the inverted potential. 



The solution of © supplemented with the boundary conditions 
m(z) — m-(h) for z £ {— L, —L + w — 1} and m(z) — 
m + (h) for z € {L — w + 1, ...,£} yields the magnetization 
profile of the chain. Once this profile is known one can obtain 
the Van der Waals curve by inverting the relation 



1 L 

Em(z) 

z — — L 



(3) 



As shown in figure [3] this curve has the same characteristics 
as the GEXIT curve of convolutional LDPC codes. As L 
increases the curve becomes almost equal to the Maxwell 
isotherm, already for w = 1. The wigles correspond to 
transitions between stable kink states (see top left in figure 
|U). As w increases one observes that the amplitude of the 
wigles diminishes rapidly. 

Figure [4] depicts the profiles m(z) obtained for L = 25, 
w = 1, J = 1.1. For h — one finds approximately 50 stable 
kink solutions of finite width with a center fixed by m as it 
varies in the region of the wiggles; the figure shows a kink 
corresponding to m — 0. For h = 0.55 the stable solution of 
the individual system constitutes the major part of the profile. 

1) A mechanical analogy: The nature of the solutions m(z) 
can be understood qualitatively as follows. Defining $/j(m) = 
— $(m) + hm relation d2j can be rewritten as 



J D 2 



4 Dz 2 



dm{z) 



<& h (m{z)) . 



-m(z) 

z 

This is a discrete Newton equation for a ball of mass j 
rolling in an inverted potential ^(m) (see figure |4). Here 
z € {—L, ...,£} is time and m(z) is the position of the 
ball. We do not solve the equation of motion for fixed initial 
position and velocity as is usual in mechanics, but rather 
for given boundary conditions: we ask that at time — L the 
ball is at position rn_ (on the left maximum of the inverted 



potential) and at time +L it ends up at position m+ (on the 
right maximum). For h = the initial and final potential 
energies are the same so that the ball must start and finish 
with infinitesimal velocity. The ball departs from the left 
maximum after an infinite amount of time, rolls through the 
minimum in a finite amount of time, and then climbs to the 
right maximum in an infinite amount of time. The instant of 
departure of the ball is determined by the "average position" 
m, equ. ©. On the other hand for h ^ the ball starts on 
the left with finite velocity such that its initial kinetic energy 
exactly equals the difference in potential energy between the 
two maxima, because it has to end up on the right with zero 
velocity. The ball immediately departs from the left, rolls 
through the minimum in a finite amount of time, and then 
climbs towards the right maximum in an infinite amount of 
time. This picture can be confirmed by an explicit analytical 
solution of the continuum version where time z is continuous 
and the finite difference operator is replaced by a second 
derivative with respect to time. This calculation yields the 
following aproximation for the profile when h = 0, 



m(z) « v^(7^T)tanh{^^-il(^ + 



)}■ 



This can be shown to be exact in an appropriate scaling 
limit. For h ^ one can also derive separate analytical 
approximations valid for z « — L and z >> — L. 

B. Chain of Random-Field Curie Weiss Systems 

When an LDPC code is viewed as a spin system the channel 
outputs play the role of a random magnetic field that is 
added to the Hamiltonian. This is our motivation to consider 
the RFCW model and check that the general picture of the 
previous paragraph still holds. The RFCW model is defined by 
adding a contribution — J2iz Hiz s iz to the Hamiltonian where 
Hi Z are i.i.d random variables with a well behaved symmetric 
density E[Hi Z ] = 0. A generalized form of coupled Curie- 
Weiss equations determines a profile for the expected value 
of the magnetization and a Van der Waals curve can again 
be defined from (01. Numerical solutions display the same 
features as in the deterministic case. We do not give more 
details here due to lack of space. 

IH. Coupled Constraint Satisfaction Models 

We now turn to a much more challenging class of mod- 
els, namely constraint satisfaction models defined on random 
sparse graphs, and focus on two paradigms, if-satisfiability 
and Q-coloring. Both models display a static sat-unsat phase 
transition as a function of a control parameteiQ p: for p < p s 
one finds exponentialy many solutions satisfying all con- 
straints (sat phase), while for p > p s no such solutions 
exist (unsat phase). This is the prediction of the zero tem- 
perature cavity method applied to locally tree like graphs 
0. Within this formalism the sat and unsat phases are the 

'As we will see, for if -SAT p = a the clause density and for Q-COL 
p = c where c/N is the probability that an edge is present. 



globaly stable solutions of a set of fixed point equations called 
survey propagation (SP) equations. SP equations are a set 
of message passing equations - which may also be viewed 
as BP equations associated to a "derived graphical model" 
- where the messages are probabilities attached to the edges 
of the graph. The sat phase is associated to a trivial fixed 
point solution^ that exists for all p. To find the other solutions 
one solves the SP equations iteratively and it turns out that 
when p crosses psp (psp < Ps), called the SP threshold, non 
trivial fixed points appear. In [psp,Ps] me geometry of the 
space of solutions is non trivial and may display various other 
thresholds, a question that we do not adress here. 

We will show, both numericaly and analyticaly (in large K 
and Q regimes) that coupling again leads to the boosting of 
the message passing threshold j>g p uplcd — > p s . As expected 



one also finds p' 



coupled 



p s . It is natural to conjecture 



that other thresholds of the coupled model that may exist in 

^coupled ^coupled] a j SQ tend tQ ^ 

A. K -Satisfiability 

1 ) General formalism: Consider a general but fixed bi- 
partite graph G with variable nodes {i,j, ...}, clause nodes 
{a, b, ...}, and edges connecting only variable to clause nodes. 
Edges come in two versions, dashed or full. Dashed edges are 
marked with Ji a = 1 and full ones with J ja = —1. The set 
of neighboring nodes of a (resp. i) is called V a (resp. Vi). 
Boolean variables Xi G {0, 1} are assigned to variable nodes 
and a clause a is said to be satisfied iff Viev a x t ia — 1' where 
xf 1 = Xi and x~ x = a^. A satisfying assignement is one 



1. 



that satisfies all clauses simultaneously A a (V 'iev a x i 
Note that this model can easily be cast in Hamiltonian form 
in terms of Ising spins. 

We briefly explain the content of SP message passing equa- 
tions. Here we adopt the formalism of [8|. If a is unsatisfied 
by all variables in V a \ i then it sends a "warning" to i with 
probability r\ a ^i. This is computed from other probabilities 
sent to a by nodes j £ V a \i- Consider the warnings received 
by j from clauses b G Vj \ a. These come in two categories: 
those that are "impeding a" and those that are "supporting 
a". A warning received by j impedes a when the edges 
(b — > j ; j — > a) are (dashed; full) or (full; dashed) because 
j will obviously not be able to satisfy both a and b. On the 
other hand if the edges (b — > j ; j — > a) are (dashed; dashed) 
or (full; full) the warning received by j supports a. Let ir^ a 
(resp. 7Tjl^ a ) be the probabilities that j receives no supporting 
(resp. no impeding) warning. We have: 



Va- 



n 



jev a \i n j- 



The two disjoint sets (resp. V ia ) are such that edges (b — > 
j ; j — > a) are of the same (resp. different) type. 

2 Strictly speaking one looks at the fixed point equation satisfied by the 
probability densities of the messages. 



2) Coupled K-SAT: We apply this formalism to graphs that 
are instances of a coupled ensemble. At each position z = 
—L — w + 1, ... + L + w — 1 we lay down sets of clauses 
a z and variable nodes i z whose cardinalities have ratio a. 
Each clause a z has K emanating edges. Each edge connects 
to a node at position z + k, call it i z +k, where fc is picked 
uniformly at random in 0, ...,w — 1 and then i z+ k is picked 
uniformly at random from nodes at the position z + k. An edge 
is then turned into dashed or full with probability i. Note that 
the degree of the clauses is equal to K and the one of the 
variable nodes is Poisson^JT). Note also that for w — 1 we 
recover the usual uncoupled K-sat graphs. 

The distributional equations associated with the message 
passing system are solved iteratively by a sampled density 
evolution or population dynamics method. It is convenient 
to work with the entropic variables <\> = — ln(l — r\) and 
x^ — — In 7T^~ . There always exist a trivial fixed point ~E{(f>] 
= corresponding to a vanishing "warning entropy". Below a 
treshold agp Upled this is the unique fixed point, and non trivial 
solutions for the warning entropy appear above this threshold. 
Let us list some of our numerical observations. For K = 3 



we have asp = 3.93, a s = 4.266 and a 
3) = 4.270, a SP uplcd (I 
we have asp = 8.3, a 



40, w 



coupled 
SP 



(L = 30, w 



3) = 4.268. For K — A 

coupled^ _ 3Q w = 



9.931 and a™ upiea (L 



3) = 9.935, asp Uplcd (X = 50, w = 3) = 9.932. The message 
passing threshold of the chain comes very close to a s already 
for small values of w. 

3) Large K -limit: For the uncoupled system in this limit 
a s = 2 K \n2 - and a S p = ^{InK + In In K + 1 - 

ln2 + 0(is^)) |0. Thus we set a = 2 K a, and take K large 
while w and L are kept fixed. In terms of the entropic variables 
the message passing equations become sums with O(K) terms 
and one can reasonably assume that their distibution is picked 
on their average values. Using this approximation we find for 
z G -L, L 



f(z) = aK 



1 



E 



k=o e m ^3=° 



-j+k) 



K-l 



where ip(z) = 2 K ~ 1 aK~E[cf) z } is the scaled warning entropy 
emanating from nodes at position z. For w = 1 we get the 
equation satisfied by the individual uncoupled system, namely 
if = aK{ } K ~ 1 . This equation is the analog of the Van 
der Waals curve ((TJ (or EBP GEXIT function), and serves to 
fix the boundary conditions in the set of coupled equations. 
Given a, for z ^ {— L, ...,£} we fix ip(z) to the trivial fixed 
point solution on the left and to the other non-trivial globaly 
stable fixed point on the right. The Van der Waals curve for 
the coupled system is given here by the plot of a as a function 
of Tp where Tp = 2L 1 +1 Ylx=—L 'P* ( see figure |5] for numerical 
solutions). 



B. Q-Coloring 

In the Q-coloring problem one assigns colors among 
{1,...,Q} to all the vertices of a graph. Assignements such 
that any pair of adjacent nodes have different colors are called 




Fig. 5. VdW curves for 5-SAT. Vertical axis is a and horizontal is 
the average warning entropy. Left: uncoupled 5-SAT; the upper red 
line is at a s = In 2 — 1+ <^ 2 w 0.6666 and the lower green line 
is at asp ~ 0.5129. Right: (L — 50, w = 3). In the circle the 
vertical magnification is 10 2 . There are approximately 100 wigles 
corresponding to transitions between kink states for the profile of the 
warning entropy. As w increases their amplitude rapidly diminishes. 

satisfying assignements. The problem is equivalent to the 
Potts model Hamiltonian at zero temperature and it has been 
analyzed by the cavity method for instances of Erdos-Renyi 
random graphs [9|. This leads to a set of SP equations. 

We analyze the SP equations for an ensemble of Erdoes- 
Renyi graphs coupled into a one dimensional chain. At each 
position z € {—L, ...,£} place a collection of vertices i z G 
{1,...,N}. Now for each pair of positions such that \z% — 
zi\ < 2w (we allow z\ = z%) consider all pairs of distinct 
vertices i Zl j Z2 and connect them with an edge, with probability 
2w+i !■ Gi yen a position z in the bulk and a vertex i z there 
are di z ~ Poisson(c) outgoing edges all connected to vertices 
that are at positions in the window {z — w, z + w}. The 
case w — corersponds to the individual uncoupled system. 

The system displays phase transitions as a function of the 
graph density c with the same numerical features as in K- 
SAT. An analytical analysis for the limit of large Q confirms 
the numerical findings. 

TV. Discussion 

In the ususal paradigms of statistical mechanics, LDPC and 
other coding constructions, or random constraint satisfaction 
problems, one looks at the macroscopic behavior of large 
assemblies of interacting microscopic sytems. Here we look 
at the behavior of large assemblies of mean field macroscopic 
systems coupled into a one-dimensional chain. The equilib- 
rium phases of the individual system are forced to coexist 
by the boundary conditions. This generates the kink solutions 
that appear as stable solutions of the fixed point equations 
describing the chain. It is the stability of these solutions that 
implies the saturation of the dynamic threshold of the coupled 
ensemble to the static one of the single system. Clearly this 
mecanism can work out only if the single system has no spatial 
or finite-dimensional geometrical structure (i.e. is mean field). 
Indeed any system defined on Z d can be thought as a coupled 
chain of systems on Z d_1 ; but we know that the phenomenon 
of threshold saturation does not occur for such systems (e.g 
finite dimensional Ising models). 

Our systems are a hybrid between mean field models and 
Kac models. In Kac models degrees of freedom (e.g spins, 



atomic positions) live in a finite dimensional space and interact 
via a finite range pair potential, whose length scale is sent to 
infinity while its intensity is sent to zero in a proper way. In 
the limit, the Kac model Hamiltonian looses all trace of spatial 
structure and becomes mean field (e.g it may ressemble the 
CW Hamiltonian). But because the free energy of a system 
with local potential has to be convex, the free energy of the 
Kac model is still convex in the limit. This means that it will 
be given exactly by the Maxwell construction (or convex hull) 
applied to the one of the mean field limiting Hamiltonian. This 
was proven in the classic work iflOl . Now, the convolutional 
LDPC ensembles, as well as the other models discussed 
in this note, have Hamiltonians that are already mean field 
in the direction "perpendicular" to the chain, and have a 
Kac interaction of width 0(w) along the direction "parallel" 
to the chain. Thus one would expect that as w grows the 
GEXIT/Van der Waals curves of the coupled chain comes 
close to the Maxwell construction for the limiting model. And 
since the limiting model is "just a bigger" uncoupled model, 
the GEXIT/Van der Waals curve of the couped system should 
converge to that of the individual mean field system. These 
arguments show that threshold saturation should be expected. 

Clearly there are many open questions that are worth inves- 
tigating. Here let us just mention possible connections with 
coarse grained theories of interfaces, coupled map systems, 
discrete soliton equations, and perhaps most importantly algo- 
rithmic implications of the phenomenon of treshold saturation. 
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